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30 college students served as paid ^s in a study of the 
frequency of answer observing while doing self-lnttructlonal 
prografkimed material. 2 different programs were used. Ss In 
the experimental groups were Instructed either before doing the 
second program, or after, vthdt they would be pai:d on the baefe 
of their score on a post-program teat. Frequency of error, 
frequency of answer observing, and post- test scores were deter— 
inlncd. Sa were found to vary In their answer -looking frequencies; 
the probability of answer observing on those items (frames) which 
S had answered Incorrectly was higher than on frames which he had * 
answered correctly. The Incentive (experimental) variable did not 

have a marked effect on any of the dependent variables except post* 
test score. 



This Is a continuation of a series of studies aimed at Investigating 
those conditions in programmed teaching material which seem necessary if 
exposure to the correct answer of a frame Is to be reinforcing to the student. 
In an earlier paper (Gels et al., 1965) the widespread contentio>n that the 
answer In a text program Is reinforcing to the student was questioned 
both logically and empirically. Those studies revealed that only under certain 
conditions did students observe answers In a program; a reasonable extension of 
that finding would be that the printed answer In a program is reinforcing to a 
student only when a specific set of conditions obtain. The pilot studies ' 
indicated that the population of learners was trimodal In regard to answer 
observing: a few students looked at every answer, a few students looked ^at no 
answers, and most students looked at about 1/4 to 1/3 of the answers that were 
available for observation. The studies also Indicated that answer observing 
was related In a regular way to student error; the probability of a student 
observing an answer after having made an error on a frame was higher than 
®^^®^ having made a correct response. Furthermore, one study revealed that^ 
the student's rating of his confidence In his own answer on a particular frame 
was as good a predictor as (or, perhaps, a better predictor than) the 
correctness of the student's answer. 

The present study attempts to replicate the earlier findings and to 
iivestlgate the effect on a number of dependent varllibles of two operations which 



might be termed incentive producing. The incentive instructions were given 
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to one group before they did a second program and to a second group immediately 
after they finished the second program but before they took the post-program 
test. Thus, these twogroups might be termed the "incentive learning and the 
"incentive performance" groups. 

Apparatus and Procedure 

Mate rials. The materials used in the study were sections of two self- 
instructional programs designed as a review for physicians, the first 66 
frames of Allergy and Hypersensitivity and the first 99 frames of Physician's 
Liability (Pfizer,^ 196A, 1966). 

Each frame was mounted separately on a 5" x 8" white index card;' the answer 
to the frame was mounted on the reverse side of the card; the card was then 

enclosed in a clear plastic protector. 

Some minor changes were made ip the programs. In six frames of the 
Liability program a technical medical term referring to procedures or opera- 
tions was accompanied by a more common synonym which was penciled in next to 
the technical, word. One sentence which referred the. reader to supplementary 
material in the appendix was removed from both programs. (These were additional 
references and-, as noted in the text^ were unnecessary for learning the material 
in the program.) Frame 30 in Liability could be considered calling for an opin- 
ion; no correct answer was given in the program. The frame was left in but 
neither errors nor looking data from that frame are considered in the results. 

In the few cases of continuous text material in the programs ^ materials which 
called for no written student response, the tes'^t was divided into convenient 
segments and mounted on successive cards in the same manner as the frames. 

Accompanying each program was a blank answer sheet with item numbers and 
answer spaces duplicating those in the frame. 

In addition, the apparatus included two cardboard boxes, one marked in 
large letters CHECKED and the other, UNCHECICED. 

Three sets of frames and boxes were available for each program. Pre- 
and post-tests were constructed for the programs. The pre-test consisted of 
a 32-item vocabulary matching test in which single-sentence definitions were 
to be matched to technical terms. The definitions and terms were drawn from 
four areas: allergy, liability, linguistics, and thyroid function. 
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The AlUrqy post-test contelned 20 Items with e possible total score oi A5 
points; the UdHUty post-test had 24 items with a total score of 75 points. 

Each test consisted of three parts; constmcted answer, fill-in. and multiple 

choice questions e 

In addition each S, was given a Need A^shievaaent and Test Anxiety test, 

the data from which are not fQported in this study. 

Each S was given a short questionnaire to determine his aesdemlc back- 
ground in relevant subject matter areas and experience with programmed in- 
struction. Experimental sessions were held at the Center. Thirty Ss were 
used in the study. All were male freshmen or sophomores at the University 
of Michigan. Ss were paid in the manner described below. 

Method 

When a S reported for the first session, he was given a version of the 
Thematic Appirceptlon Test (TAT) with accompanying written instructions. 

Upon completion of the TAT each S was given the Test Anxiety Questionnaire 
(TAQ), again with written instructions. Following this he filled out the 
short academic biographical form and took the pre-test. This concluded the 
first session. The S was paid for his time at the rate of $1.25/hr. and 
informed that he would be notified for a later session if he was to be used 

again a 

In the second session, the S was seated before a table on which was a 
set of program cards and the pair of labelled boxes. He was then read the 

instructions! 

This is an experiment in programmed instruction. Each index card pre- 
sents a frame which provides information and may require you to make a response 
you will be supplied with sheets on which to write your responses. 

The answers to each frame are on the reverse side of the card. If you 
wish to check your response against the correct answer, simply flip the 
card over after you have written your answer on the answer sheet. You may 

check as many or as few responses as you wish. 

If you check an answer by flipping over the card, place the card in the 

box marked "checked." If you do not cheek the answer, place the card 
in the box marked "unchecked." If you decide to check the printed answer 
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to the frame and you find that you are Incorrect, do not go back and change 
your response. Do not go back to previous frames unless you are specifically 
Instructed to do so. You id.ll be given a post-test when you have finished. 

Are there any questions? 

Half of the went through the Liability program and half through the 

Allevjy* worked at separate tables usually in small groups of three or 

four. The time taken to complete the programs was recorded. 

Upon the completion of the program the ^ was given the post-test, one 
page, or section, at a time. Before leaving the session a time was set for 
the next session. The ^ was not paid at the end of this session, having 
been Informed that he would receive full pay at the end of his last session. 

At the beginning of Session 3, Sb were Instructed differently according 
to their grouping. Three groups hac^ been constructed, balancing each for 
distribution of TAT and TAQ scores. Each ^ In Session 3 worked on the program 
he had not done In Session 2. 

Groups I, II, and III were re-read the Instructions for Session 2. 

Group II ^s ("Incentive learning" group) were then given typed Instruc- 
tlons which had been clipped to the first pcge of their program response 
sheets: 

You have already earned $ in the previous session. Your final payment, 

for this session will depend upon how well you score on the post-test. You 

will recelye 5c for each percentage point of your score. Therefore, you can 

earn as much as $5.00 for this session. I cannot tell you any more than this. 

After completing the program. Group III £s (''incentive performance" group) 
were given typed Instructions which had been clipped to the first page post- 
test answer sheet: 

Now that you have finished the program, here are some further Instructions. 

You have already earned $ In the previous session. Your final payment 

for this session will depend upon how well you score on the post-test. You 

will receive 5C for each percentage point of your score. Therefore, you can 

earn as much as $5.00 for this session. I cannot tell you any more than this. 

Each £ went through the program, took the post-test and then was paid 
for his last two sessions. Before leaving he was given Instructions not to 
discuss the experiment with other students and was thanked for his cooperation. 
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A summary of the groups and procedures may be found in Table 1* 



Insert Table 1 about here 



Results and Discussion 

Previous research has led to the expectation that with most programs a 
small number of the frames and ^s contributes most of the errors made on frames. 

Figures 1 and 2 indicate this to be true of the two programs and the populations 
used in the present study. In Figure 1 the cumulative error is plotted against 

■ ^ 

the frames in the programs ranked for error. Thus, the first point plotted 

Insert Figure 1 about here j 

for the Liability program is the total number of errors recorded for the highest j 

error frame, the second point adds to it the total number of errors scored on 

the second highest error frame, etc. More than half of the total number of 

errors made by all Sa is accounted for by less than 1/4 of the frames in each i 

program. In Figure 2 it can be seen that about 1/3 of the Ss account for half 

of the errors recorded. 




.Insert Figure 2 about here 



The per cent error for Sa and frames is higher than that reported as 
generally acceptable in well-designed programs. However, these programs were 

S: 

designed for physicians and our test population consi&ted of college students 
with no medical training. The material was not only foreign to them (as 
revealed by low scores on pre-tests), but also the interest or motivation for 

•J 

doing the program in this population was probably based upon payment whereas 
in the target population there are professional reasons for the physician to 
master the material in the programs. Therefore, it would be a mistake to be 
critical of the programs qua programs on the basis of the high per cent error 
tnilleatad here, iisvertheless, the patterh if not the magnitude, of high error 
frames and high error ^s is typical of the data we have obtained on other programs. 

Figure 3 shows frequency plots of answer observing for each program, the 
data being presented separately for first and second program presentations. 
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Insert Figure 3 about here 

Thus, Liability la, Ila, Ilia is a figure of the data of those 15 Ss who took 
the Liability program first and the Allergy program second. In all cases the 
data are similar to those obtained in previous studies (see Figure A). Students 

Insert Figure 4 about here 

— — r~ — ^ !■ III ! r~i 

do not regularly observe all of the answers in a program. On the average, 
they looked about one-half of the time on these programs and, in the case of 
all but one group, regularly observed answers more often on the second program 
than on the first. As* in previous studies, there are Ss who never look at an 
answer in the program, ^s who always look at an answer in the program, and the 
majority are somewhere in th© middle. 

As previously mentioned, earlier studies revealed the relationship between 
student error and answer observing. Figures 5 and 6 present data relevant to 
this observation. In these figures the probability of observing an answer after 

t 

^ Insert Figures 5 and 6 about here 

having made an Incorrect response (Pw/w) is plotted against the total percentage 
of looks for each The diagonal line on each figure represents chance level 
cpf looking when wrong for all total answer- looking per cents. Thus, if a ^ 
looked at answers in the program 50% of the time, his chance level of looking 
when wrong would be 50% and his data point would fall on the line. It is 
obvious that in almost all cases the ^s, regardless ©f their total per cent 
of looking, look at answers in the program much more often when they are 
incorrect than when they are correct; almost all of the data points fall to 
the left of the Indicated chance level. The implication is, of course, that 
the _S is able to discriminate, on the basis of his own answer, something about 
the correctness of that answer and tends to check the printed answer in those 
eases in which he himself has supplied an incorrect answer. A second observa- 
tion may be made about these d©ta’— namely, that looking and looking— when— wrong 
percentages are related. Figures 5 and 6 are not good illustrations of this 
point since the total looking rate is plotted on the abscissa. However, when 
the data are analyzed into looking-when-rlght and looking— when— wrong rates, a 
clear relationship between the rates is disclosed, i.e., extremely high Pw/w 
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predicts high looking rates when correct and vice versa while low Pw/w predicts 
low Pc/d and vice versa. The frequency of looklng-when-wrong Is not a function 
of the error rate of a particular S, A low error-rate ^ may or may not have 
as good a discrimination (9 tendency to look more often when wrong than when 
right) as a high error 

It Is Interesting to note from data not graphically presented that the 
Pw/w Is generally greater for the second program than for the first. There 
Is at least an Indication here that as a student continues to work on the 
programs he tends to develop more effective or efficient strategies. The most 
efficient strategy, of co»^rse, would be to check only those answers which are 
incorrect. This indication of the acquisition of more effective learning 
strategies as* a function of more experience with* the learning materials will 
be pursued in later studies. The word "effective” is intentional, as there 
is some indication from the data that a shift upward in a student’s Pw/w is 
accompanied by an increased post-test score. 

gure 7 shows consistency of looking within ^s. A significant correla- 
tion is obtained w»an the per cent looking on a first program is plotted as 
a function of the per cent looking on the second program. Thus, a can 
truly be termed a high or low looker. 

Inserj: Figure 7 about here 



Claims have been made for the necessity of observing the answers in a 
program in order to produce effective learning. Figure 8 shows the relation- 
ship or lack of it between answer observing and post-program test score 

Insert Figure 8 about here 

using the Allergy program. No relationship is found. A simple-minded notion 
that mere exposure to the answer as well as to the frame on a program is 
important in learning must therefore be discarded. 

It has already been pointed out that the error rates are high for some 3s 
and for some frames in each program. The consistency of error within Ss is 
demonstrated in Figure 9. Here the per cent errors in the two programs are 

Insert Figure 9 about here 
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plotted. It seems safe to conclude that there are ^s who consistently maintain 
a high or low error level regardless of the program. The picture emerges from 
these observing and error data that the within-^ variables such as strategy of 
learning and competencies related to performance within the program show up 
regularly and consistently. These variables clearly deserve further study. 

Figure 10 lends some support t the conjecture that errors on programs 
are related to the amount of achievement In the program. Here the post -program 
test scores for the Allergy program are plotted against the per cent error 
within the program. The lower error ^s clearly tend to perform better on the 
test. 



Insert Figure 10 about here 

Group comparisons . Until now the data have been reported with all ^s 
considered as a single group. Table 1 presents the means for ^s In different 
groups for the various dependent variables. Groups la and Ib might be 
considered control groups. The Lioibiltty post-test scores tend to be higher 
than the Allergy post-tv^st scores. Data from the experimental groups Indicate 
that the Instructions regarding post-test contingent payment, whether those 
Instructions were given before or after the ^s went thropgh the program, tended 
to raise the post-test scores. Thus, the Allergy post-test score seems Inflated 
for Group Ila; the Liability post-test score seems Inflated for Group Ilb and 
similarly, the Increase In the test score related. to the payment Instructions 
can be found In the two sub-groups of Group III. If the difference scores 
are calculated for the Liability tkLwxs the Allergy post-tests for each group. 

It Is clear that Instructions seem to have tipped the balance In favor of the 
payment-connected post-test. It Is not unexpected that the setting of such 
explicit contingencies would produce a change in the achievement scores. The 
difference between being Instructed about the contingencies before taking the 
program or directly before taking the post-test (ths difference between 
Groups II and III) seona to be Insignificant. The difference scores for 
Group lib and Illb arn exactly the same, the scores for Groups Ila and Ilia 
are only slightly different. 

Of greater Interest In this study was the effect of instructions on 
answer observing. It predicted that Group II Ss would tend to observe 
more answers than Groups I or III. The simple but logical hypothesis was that 
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the Instructed ^s in Group II might tend to be ^ore cautious and develop a 
strategy which involved a higher rate of answer observing. Again, difference 
scores in observing suggest that Group II did observe relatively more on the 
program following instructions than would be expected, extrapolating from the 
Group I data. However, the data from Group Illb casts a shadow upon any 
conclusion concerning the effectiveness of instructions. 

Data from the control group suggest that answer observing is a function 
of interaction between the program content and whether the program is taken 
first or second. The control group shows that the L%ob%X%'ty ptogram tends to 
produce more answer observing. Thus, the difference scores between answer 
observing for LidbiZity and AZZergy for Group la is -.6 and for Group Ib, 
+10.0. The tentative conclusion is that the Liability program produces 
relatively more observing behavior than the Allergy program when the factor 
of order is removed. The prediction might then be that for the sub-groups in 
Group II these tendencies would be exaggerated. Thus, in Group Ila the 
Allergy program was in second position as well as the program before which 
instructions were given. Therefore, the difference between observing on 
Allergy and Lioibility should be increased in favor of Allergy* This is indeed 
what occurred. In Group Ilb the Liability program was second and also was the 
pre-instructed program; the difference should be exaggerated in favor of the 
Liability program. In this case, the difference score is not as great as in 
Group Ib. In the sub-groups of Group III there is no reason to expect data 
that differ markedly from those of Group I and yet they do. Group Illb 
especially shows an unezspected reversal with the second program (J/idb'iltty) 
having less answer observing than the first program {Allergy) . There is some 
reason to suspect that Group III' ;as an atypical collection of But 

rather than attempt to argue away the data, it should merely be noted that 
there is little support for any conclusion which suggests the instruction 
variables had a marked effect on answer observation. 

With regard to errors oi.. the program, the error rate c>n the Liability 
program is consistently higher than the error rate on the Allergy program 
for all groups. When the difference scores are determined, by subtracting 
the error rate for Allergy from the error rate for Liability^ again a 



429 



10 



Gels 

comparison can be made between instructed and non-instpucted groups. It might 
be expected that Ss in Groups Ila and Ilb would show lower error scores because 
of the care with which they would go through the program, . knowing ahead of 
time that their post-test scores would determine their payment. Thus, in 
Group Ila the expected smaller AlUm error rate should be even further 
reduced while in Group lib the higher Liability error rate should be reduced. 
Again, the trend is in the right direction with an increase in the difference 
score from Group Ila and a decrease in the difference score for Group lib, 
but the variability of the data precludes any conclusion about the effect of 
the instructional variable on the error rates within the program. 

Sumtnary and Conclusion * 

The study has replicated and supported earlier indicative data concerning 
the variability in rates of observation of answers in a progrm, and the 
relationship between answer observation and error in programs.' In addition, 
it has^ provided data which indicate there is wlthln-S, consistency in terms 
of error and in terms of answer observing. The present study supports the 
observation previously made on the basis of earlier studies that the answer 
in a program cannot automatically be considered reinforcing but rather may 
act as a reinforcer only under certain conditions. 
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